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ABSTRACT 

Embedded  Training  Systems  provide  a  new  opportunity  for  training  not  only  procedural  or  systems  skills, 
but  also  the  higher-level  cognitive  skills  that  are  critical  for  successful  performance  on  today's  battlefield. 
We  have  developed  a  principled  approach  for  training  the  essential  cognitive  skills  underlying  high  levels 
of  situation  awareness  (SA)  and  decision  making  that  can  be  easily  employed  in  future  embedded  training 
systems  so  as  to  significantly  enhance  this  critical  commodity.  These  training  approaches  include  (1)  SA 
Trainer  which  employs  systematic  experiential-based  training  to  build  essential  knowledge  bases  and 
higher-order  cognitive  skills ,  (2)  Virtual  Environment  Situation  Awareness  Review  System  (VESARS) 
which  employs  informed  feedback  on  SA  skills,  team  communications,  and  the  accuracy  and  completeness 
of  S A,  and  (3)  the  SA  Virtual  Instructor  (SAVI)  which  builds  critical  SA  skills  by  calibration  of  good  and 
poor  SA  practices. 


1.0  INTRODUCTION 

New  systems  being  built  for  C4ISR  and  other  battlefield  operations  increasingly  include  embedded 
training  capabilities  -  that  is  training  that  occurs  on  the  actual  systems  and  devices  the  soldier  normally 
uses  to  perform  his  or  her  duties.  While  some  of  this  training  may  take  the  form  of  more  traditional 
Interactive  Media  Instruction  (IMI),  it  also  involves  simulation  based  training  in  which  actual  operations 
can  be  simulated  (with  either  live  or  virtual  team  mates  and  opponents)  and  the  soldier  being  trained  can 
interact  with  the  controls  and  displays  of  the  system  in  the  same  way  he  would  in  the  real  world.  This 
train-as-you-fight  concept  has  considerable  advantages  in  terms  of  costs,  time  utilization  and  opportunities 
for  better  training  outcomes. 

Embedded  training  is  likely  to  be  used  for  training  soldiers  in  system  interaction,  and  in  military  doctrine 
and  procedures.  The  real  advantages  of  embedded  training,  however,  go  far  beyond  these  standard  training 
objectives.  Embedded  training  systems  also  provide  the  perfect  platform  for  providing  the  enhanced 
cognitive  skills  training  that  is  critical  in  the  future  battlefield.  In  particular,  we  have  been  involved  in 
ongoing  work  using  these  types  of  virtual  simulations  to  train  situation  awareness  (SA)  skills  and 
knowledge  in  military  settings.  SA  is  the  key  cognitive  construct  upon  which  decision  making  rests  and 
one  of  the  most  challenging  to  develop  and  maintain  in  dynamic  environments  such  as  combat.  These 
training  approaches  are  based  on  research  that  shows  the  key  challenges  for  SA  and  research  on  factors 
that  distinguish  experts  from  novices  in  these  domains.  They  are  also  built  on  detailed  cognitive  task 
analyses  in  the  military  domain  settings  that  reveal  the  critical  SA  elements  for  a  given  warfighter  role  and 
the  challenges  they  must  contend  with. 
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1.1  Cognitive  Skills  Underlying  High  Levels  of  Situation  Awareness 

In  talking  about  situation  awareness,  we  encompass  all  three  levels  of  SA  in  our  approach.  Situation 
Awareness  is  “ the  perception  of  the  elements  in  the  environment ,  within  a  volume  of  time  and  space ,  the 
comprehension  of  their  meaning,  and  the  projection  of  their  status  in  the  near  future”  [3].  It  therefore 
includes: 


•  Level  1  SA — Perception  of  the  elements  in  the  environment.  The  first  step  in  building  SA  is  to 
perceive  the  status,  attributes,  and  dynamics  of  relevant  elements  in  the  environment.  This  will 
include  important  elements  such  as  enemy,  civilian,  and  friendly  position  and  actions,  terrain 
features,  obstacles,  and  weather.  Inherent  in  military  operations  is  the  difficulty  associated  with 
determining  all  the  needed  aspects  of  the  situation  due  to  obscured  vision,  noise,  smoke, 
confusion,  and  the  dynamics  of  a  rapidly  changing  situation.  Additionally,  the  enemy  works  hard 
to  deny  information  on  their  troops  and  operations  or  to  intentionally  provide  misleading 
information. 

•  Level  2  SA — Comprehension  of  the  current  situation.  Comprehension  of  the  situation  is  based  on 
a  synthesis  of  disjointed  Level  1  elements.  Level  2  SA  goes  beyond  simply  being  aware  of  the 
elements  that  are  present  to  include  an  understanding  of  the  significance  of  those  elements  in  light 
of  the  warfighter’s  goals.  This  level  of  SA  is  sometimes  called  situation  understanding  (SU).  The 
warfighter  assimilates  Level  1  data  to  form  a  holistic  picture  of  the  environment,  including  a 
comprehension  of  the  significance  of  objects  and  events.  For  example,  an  intelligence  officer  may 
need  to  assimilate  the  data  from  multiple  sensors  and  reports  to  determine  enemy  intent  or  the 
impact  of  friendly  operations  on  the  degree  to  which  an  enemy  unit  can  shoot,  move,  and 
communicate.  Typically  Level  1  SA  (perceived  data)  must  be  interpreted  (with  reference  to  goals 
or  plans)  in  order  to  have  meaning  as  Level  2  SA.  For  example,  a  commander  must  understand  the 
impact  of  discovering  a  new  enemy  unit  on  the  conduct  of  a  mission  operation,  so  that  he  can 
rapidly  make  needed  adjustments  to  ensure  its  success 

•  Level  3  SA — Projection  of  future  status.  The  third  and  highest  level  of  SA  is  the  ability  to  project 
the  future  actions  of  elements  in  the  environment,  at  least  in  the  very  near  term.  This  is  achieved 
through  knowledge  of  the  status  and  dynamics  of  the  elements  and  a  comprehension  of  the 
situation  (both  Level  1  and  Level  2  SA).  Commanders  with  a  very  high  level  of  SA  are  able  to 
project  where  and  when  the  enemy  will  strike  or  what  their  next  move  might  be.  Warfighters  with 
Level  3  SA  are  able  to  project  how  much  time  they  have  until  reinforcements  arrive  or  until  a 
second  volley  of  artillery  fire  arrives.  This  gives  them  the  knowledge  and  time  necessary  to  decide 
on  the  most  favourable  course  of  action  to  meet  their  objectives 


1.1.1  Challenges  for  Novices 

Expertise  in  a  particular  domain 
has  a  significant  role  in  allowing 
people  to  develop  and  maintain  SA 
in  the  face  of  high  volumes  of 
information  transfer  and  system 
complexity  [4].  This  can  best  be 
explained  in  terms  of  two 
divergent  ends  of  a  continuous 

spectrum,  as  shown  in  Figure  1 . 


SA  is  demanding, 
frequently  incomplete 
and  erroneous 

A _ 


Novice 


•  Limited  attention 

•  Limited  working  memory 


SA  is  fast,  can  be  effortless, 
more  complete,  greater 
comprehension  &  projection 

_ ^ 


Expert 


V 

•  Schema  of  prototypical  situations 

•  Mental  modelsof  domain 

•  Automaticity  of  processes 

•  Learned  skills  (e.g.,  scan  patterns, 
communications) 


Figure  1.  Factors  effecting  SA  in  Novices  and  Experts  in  a  Domain  [1] 


A  person  who  is  completely  new  to 

the  systems  and  situations  in  a  particular  domain  (e.g.  a  brand  new  first  lieutenant  or  a  soldier  who  is  new 
to  a  combat  zone)  will  be  considerably  overloaded  in  seeking  to  gather  information,  understand  what  it 
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means  and  formulate  correct  responses.  They  will  be  severely  hampered  in  their  efforts  by  both  limited 
attention  and  limited  working  memory.  Lacking  other  mechanisms,  they  will  have  to  think  through  each 
piece  of  data  and  try  to  process  it  in  working  memory  along  with  other  pieces  of  data.  Reading  each  report 
or  listening  to  radio  communications  and  interpreting  that  data  will  impose  a  significant  burden.  Properly 
understanding  the  significance  of  what  is  perceived  will  likely  be  problematic  as  well,  as  they  will  not 
have  the  experience  base  on  which  to  interpret  that  information.  So  they  will  often  remain  far  behind  the 
demand  curve  in  taking  in  and  processing  the  information  that  forms  the  basis  for  good  SA  in  this  dynamic 
environment. 

Their  mental  processes  are  further  compromised  by  inefficiency.  They  lack  the  knowledge  of  when  which 
information  is  really  most  important.  Scan  patterns  tend  to  be  sporadic  and  non-optimal.  They  may 
neglect  certain  key  information  or  over-sample  other  information  unnecessarily.  This  is  not  just  a  matter 
of  needing  to  learn  a  set  way  of  taking  in  information.  Without  knowledge  of  the  underlying  relationships 
between  entities  in  the  battlefield,  they  do  not  realize  what  information  to  seek  out  following  receipt  of 
other  information.  Thus  the  prototypical  novice  is  quickly  overloaded,  inefficient,  and  error  prone  in 
developing  SA.  Decision-making  and  performance  are  highly  compromised  as  a  result.  This  is  often  the 
case  reported  today  for  new  pilots  or  platoon  leaders,  for  example,  or  for  commanders  facing  a  new  kind 
of  insurgent  or  weapon  system.  Luckily,  development  of  expertise  in  a  particular  domain  significantly 
reduces  these  problems  through  a  number  of  mechanisms. 


1.1.2  Mental  Models 

With  increasing  experience  in  a  particular  domain  the  model  also  details  how  people  are  able  to  develop  a 
number  of  mechanisms  that  help  to  overcome  these  significant  hurdles.  The  first  of  these  is  a  mental 
model  of  the  systems  being  operated  and  the  operational  domain.  For  example  a  pilot  develops  not  only  a 
mental  model  of  how  the  aircraft  operates,  including  its  many  subsystems  and  its  aerodynamic 
performance  in  the  physical  environment,  but  also  a  mental  model  of  flight  operations  including  air  traffic 
control  (ATC)  procedures  and  expected  behaviour  associated  with  interacting  with  ATC  and  with  other 
pilots. 

Mental  models  have  been  succinctly  defined  as  “ mechanisms  whereby  humans  are  able  to  generate 
descriptions  of  system  purpose  and  form,  explanations  of  system  functioning  and  observed  system  states, 
and  predictions  of  future  states ”  [5].  They  are  generally  used  to  describe  a  person’s  representation  of 
some  physical  system  (e.g.  how  an  engine  or  computer  works),  but  also  can  concern  other  types  of 
systems  (e.g.  how  the  enemy  operates  or  how  the  military  decision  making  process  works).  Mental 
models  embody  stored  long-term  knowledge  about  these  systems  that  can  be  called  upon  to  direct  problem 
solving  and  interaction  with  the  relevant  system  when  needed.  Mental  models  are  highly  useful  in  the 
process  of  developing  SA  [3,  4]  and  provide  several  advantages: 

•  Knowledge  regarding  which  aspects  of  the  environment  are  relevant  and  critical  cues  that  have 
significance  (e.g.  a  flattened  area  of  grass  or  a  hum  of  an  engine).  This  knowledge  is  critical  for 
directing  attention  in  taking  in  and  classifying  information  in  the  perception  process,  making  that 
process  much  more  efficient  in  the  face  of  a  large  amount  of  information  to  potentially  be 
processed. 

•  A  means  of  integrating  various  elements  to  form  an  understanding  of  their  meaning  (level  2  SA). 
Understanding  the  significance  of  perceived  system  information  is  often  very  difficult  without  a 
mental  model.  Mental  models  provide  the  basis  for  interpreting  perceived  information  (singularly 
or  together)  in  terms  of  the  individual’s  goals  to  form  Level  2  SA. 

•  A  mechanism  for  projecting  future  states  of  the  environment  based  on  its  current  state  and  an 
understanding  of  its  dynamics  (level  3  SA).  Similarly,  the  projection  of  future  system  states  is 
very  difficult  without  a  mental  model.  Accurately  projecting  what  the  enemy  in  a  battle  is  likely  to 
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do  requires  a  very  detailed  mental  model  of  not  only  the  battlefield  (terrain  features,  weather, 
obstacles),  but  also  of  the  enemy  (objectives,  capabilities,  doctrine,  culture,  tactics,  techniques  and 
procedures). 

1.1.3  Schema  of  Prototypical  Situations  or  Patterns 

Over  time,  people  will  encounter  many  situations  and  from  this  will  develop  a  type  of  library  of 
prototypical  situations,  or  schema,  in  memory.  Schema  can  be  thought  of  as  prototypical  states  of  the 
mental  model  (i.e.  a  pattern  consisting  of  the  state  of  each  of  the  relevant  elements).  By  pattern  matching 
between  the  current  situation  and  schema,  people  can  instantly  recognize  known  classes  of  situations. 
“ Oh ,  this  is  just  like  what  happened  last  month  These  prototypical  situations  can  be  learned  through 
direct  experience  or  vicariously  through  formal  training  or  the  case  studies  and  storytelling  that  are 
endemic  in  many  professions.  “This  is  just  liked  what  happened  to  the  aircrew  in  the  Azores  accident 
A  critical  feature  of  schema  is  that  new  situations  need  not  be  exactly  like  previously  encountered 
situations  to  achieve  a  match.  Rather  only  a  few  critical  cues  may  be  required  to  lead  to  a  match  or  a  near 
match. 

Pattern-matching  to  learned  schema  provides  a  considerable  short  cut  for  SA  and  decision-making.  Rather 
than  processing  data  to  determine  level  2  or  3  SA  (requiring  working  memory  or  exercising  the  mental 
model),  that  information  is  already  a  part  of  the  schema  and  must  merely  be  recalled.  Klein  has  called  this 
recognition  primed  decision  making  [6].  The  critical  factors  for  achieving  good  SA  through  this 
mechanism  are  recognizing  the  critical  cues  that  are  used  for  pattern  matching,  and  having  a  good  stock  of 
such  schema  in  memory.  In  addition,  it  appears  that  some  people  are  better  at  pattern  matching  than 
others,  and  this  attribute  has  been  shown  to  be  correlated  with  SA  [7]. 

Very  often  scripts,  set  sequences  of  actions, 
have  also  been  developed  for  these  schema,  so 
that  much  of  the  load  on  working  memory  for 
generating  alternate  behaviours  and  selecting 
among  them  is  also  diminished.  These 
mechanisms  allow  the  commander  to  simply 
execute  a  predetermined  action  (e.g.  follow 
doctrine  or  rules  of  engagement)  for  a  given 
recognized  class  of  situations  (based  on  their 
SA).  The  current  situation  does  not  even  need 
to  be  exactly  like  one  encountered  before  due  to 
the  use  of  categorization  mapping  —  as  long  as 
a  close-enough  mapping  can  be  made  into 
relevant  categories,  a  situation  can  be 
recognized,  comprehended  in  terms  of  the 
model,  predictions  made  and  appropriate 
actions  selected.  In  that  people  have  very  good 
pattern  matching  abilities,  this  process  can  be  almost  instantaneous  and  produce  a  much  lower  load  on 
working  memory  making  high  levels  of  SA  possible,  even  in  very  demanding  situations. 

1.1.4  Critical  Skills 

In  addition  to  the  key  knowledge  constructs  of  mental  models  and  schema,  experts  also  tend  to  develop 
superior  skills  for  acquiring  and  processing  information  in  their  environment  that  must  be  learned  or 
trained.  Pilots  learn  to  communicate  with  ATC  and  to  scan  their  instruments  to  make  sure  their  knowledge 
of  the  situation  does  not  get  out  of  date.  Air  traffic  controllers  learn  how  to  scan  their  radar  maps 
efficiently  based  on  traffic  patterns  in  the  sector.  Military  officers  learn  how  to  gather  and  disseminate 


Prototypical  &  Expected: 
Schema  *  Objects 

•  Scenes 

•  Order  of  Events 


What  information  is  attended  to 
How  information  is  interpreted 
and  integrated 
What  projections  are  made 


Figure  2.  Schema  and  Mental  Models  Significantly 
Facilitate  the  Development  of  SA  [2] 
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key  information  on  the  radio,  how  to  listen  for  information  that  is  relevant  to  them  while  ignoring  other 
information,  and  where  to  post  troops  as  listening  and  observation  posts  to  insure  that  key  information  is 
gathered  and  passed  on  in  the  first  place.  Much  of  the  gathering  of  SA  is  not  just  passive,  waiting  for  key 
information  to  be  presented,  but  is  an  active  process.  The  information  that  is  available  to  military  pilots, 
for  example,  is  dependent  on  how  they  setup  and  operate  their  radar  (search  patterns,  where  it  is  focused, 
modes),  what  frequency  they  tune  their  radios  to,  and  when  they  request  information  from  others.  Thus 
their  actions  determine  what  information  they  will  obtain.  All  of  these  are  examples  of  learned  skills, 
specific  to  each  domain,  that  improve  with  expertise  and  that  contribute  to  higher  SA  among  experts. 

1.1.5  Study  of  SA  Training  Needs  in  Infantry  Operations 

As  an  example  in  the  military  domain,  Strater,  Jones  and  Endsley  [8]  surveyed  43  individuals  who  train 
new  platoon  leaders  on  problems  observed  with  SA  in  these  new  officers.  Issues  that  were  rated  as  a 
frequent  problem  by  more  than  25%  of  the  respondents  are  shown  in  Table  1.  Problems  in  communication 
posed  a  major  problem  for  new  platoon  leaders  leading  to  poor  SA.  These  problems  ranged  from  not 
requesting  information  to  not  communicating  key  information.  In  addition  there  were  significant 
problems  reported  in  gathering  information  on  the  combat  readiness  status  of  the  opposing  force  and  one’s 
own  troops. 

Comprehension  problems  were  also  noted,  including  failing  to  assemble  bits  of  information  together  into  a 
coherent  picture  and  not  specifying  alternate  courses  of  action  (CO As),  along  with  not  understanding  task 
priorities,  the  impact  of  load  and  travel  on  fatigue,  and  soldier  positioning  to  minimize  fratricide. 
Instructors  rated  understanding  enemy  strengths  and  weaknesses,  likely  areas  of  strategic  significance  to 
the  enemy,  and  enemy  expectations  of  friendly  actions  as  major  problem  areas  for  SA  in  new  platoon 
leaders. 

With  regard  to  Level  3  SA,  a  lack  of  contingency  planning  was  identified  as  a  problem  area,  as  was  failure 
to  project  the  usage  rate  of  ammunition  and  supplies.  Problems  with  SA  regarding  the  opposing  force 
were  also  found  at  the  projection  level,  as  trainers  noted  that  new  platoon  leaders  have  difficulty  projecting 
a  likely  enemy  COA,  as  well  as  their  disposition  around  heavy  weapons. 

Poor  skills  for  gathering  information  are  particularly  problematic  in  military  settings,  as  very  little 
information  presents  itself  otherwise.  It  was  found  that  the  officers  often  were  not  communicating  key 
information  because  they  assumed  they  knew  what  was  going  on,  or  that  the  information  did  not  need  to 
be  passed  on.  In  addition,  it  appears  that  novice  platoon  leaders  suffer  greatly  from  not  having  good 
mental  models  or  schema.  They  are  quickly  overwhelmed  by  information,  are  slow  to  grasp  which 
information  is  important  and  where  to  look  for  important  follow-up  information.  Without  these  schema, 
they  also  faired  poorly  in  integrating  information  to  understand  its  significance  and  to  carry  out  projection 
and  contingency  planning.  These  findings  are  well  in  line  with  the  types  of  SA  problems  experienced  by 
those  in  other  domains  [1],  and  are  indicative  of  the  areas  that  future  SA  training  programs  should  address. 


Table  1.  SA  problems  for  new  platoon  leaders  (%  reporting)  [8] 


Level  1  SA:  Failure  to  Correctly  Gather/Detect  the  Critical  Information  in  the  Situation 

Not  detecting  information  due  to  attentional  narrowing 

27% 

Not  utilizing  a  standard  reporting  procedure 

30% 

Not  carrying  out  standard  operating  procedure 

28% 

1  Poor  intelligence  information  due  to: 

Not  requesting  pertinent  intelligence 

31  % 

Not  employing  squads  tactically  to  gather  needed  information 

30% 

Not  determining  reliability/timeliness  of  intelligence  information 

26% 

Poor  communication  caused  by: 

Not  requesting  information  from  squad  leaders 

|  30%  | 
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Not  requesting  information  from  commander 

30% 

Not  communicating  key  information  to  commander 

35  % 

Not  communicating  key  information  to  squad  leaders 

30% 

Not  communicating  key  information  to  other  platoons 

44% 

Not  monitoring  company  net 

28% 

Not  communicating  overall  situation/Commander’s  Intent  to  squads 

28  % 

Not  determining  own  combat  readiness  status 

Experience  and  training 

26% 

Timing/location  of  direct/indirect  fire  support 

30% 

Not  determining  combat  readiness  status  of  opposing  forces 

Number  and  severity  of  casualties 

37% 

Physical  fatigue 

30% 

Mental  fatigue 

31  % 

Movement  and  current  position  of  troops 

28  % 

Weapons  types,  characteristics  and  quantities  available 

33  % 

Location  of  direct/indirect  fire  support 

44% 

Ammo  and  supplies  availability 

33  % 

Availability  of  reinforcements 

37% 

Heavy  weapons  location 

40% 

Past  behaviour  and  tactics 

26% 

Impact  of  current  and  future  weather  factors 

26% 

Level  2  SA:  Failure  to  Comprehend  the  Situation  (although  basic  information  is  detected) 

Not  assembling  bits  of  information  together  to  form  a  coherent  picture 

29% 

Not  specifying  alternate/supplemental  plans/courses  of  action 

32% 

Not  developing  an  understanding  of: 

Task  priorities 

33  % 

Impact  of  soldier  load  and  distance  travelled  on  troop  fatigue 

33  % 

Positioning  soldiers  to  minimize  the  risk  of  fratricide 

25  % 

Enemy  strengths  and  weaknesses 

29% 

Likely  areas  of  strategic  significance  to  enemy 

27% 

Enemy  expectations  of  friendly  actions 

34% 

Level  3  SA:  Failure  to  Project  the  Future  Situation  (though  current  situation  is  understood) 

Lack  of  contingency  planning 

39% 

Failure  to  project  the  following: 

Usage  rate  of  ammunition  and  supplies 

36% 

Likely  enemy  COA  from  available  information 

33  % 

Location  of  enemy  troops  around  heavy  weapons 

32% 

Failure  to  Effectively  Perform  the  Necessary  Mission  Tasks 

Poor  mission  planning 

27% 

Poor  responses  to  unexpected/unplanned  events 

36% 

Poor  time  management 

45  % 

Poor  task  prioritization 

28% 

2.0  TRAINING  APPROACHES  FOR  IMPROVING  SITUATION  AWARENESS 

We  have  been  building  a  set  of  training  approaches  for  improving  the  cognitive  skills  and  building  the 
cognitive  structures  underlying  high  levels  of  SA.  These  training  approaches  selectively  target  those  skills 
and  structures  that  are  specifically  needed  in  a  given  domain,  based  on  both  the  cognitive  model  described 
above,  as  well  as  detailed  analyses  of  the  SA  requirements  and  challenges  that  exist  in  each  domain.  These 
approaches  are  not  a  “one  size  fits  all”  solution,  but  rather  an  integrated  set  of  tools  that  can  be  customized 
for  the  domain,  the  needs  of  the  trainees  in  that  domain,  and  the  objectives  and  operational  realities  of  the 
organization. 
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2.1  SA  Trainer 

The  SA  Trainer  employs  rapid  experiential  learning  in  support  of  mental  model  &  schema  development. 
In  normal  operations,  over  the  course  of  many  months  and  years,  individuals  will  gradually  build  up  the 

experience  base  to  develop  good  mental  models  and 
schema  for  pattern  matching  upon  which  SA  most  often 
relies.  With  the  SA  Trainer,  we  work  to  boot  strap  this 
natural  process,  by  exposing  the  trainee  to  many,  many 
situations  in  a  very  short  period  of  time  using  IMI  and 
embedded  training  techniques.  This  experiential  learning 
is  very  structured,  so  as  to  help  the  trainee  get  the  most 
out  of  the  information  they  are  exposed  to. 


The  SA  Trainer  employs  realistic  mission  scenarios  with 
opportunities  for  complex  operational  decisions.  The 
target  is  to  provide  an  increased  opportunity  for  exposure 
to  a  variety  of  situations  which  supports  the  development 
of  situation-based  knowledge  stores,  to  train  recognition 
of  critical  cues  that  signal  prototypical  situations,  to  support  trainee  information  integration  and  decision 
making,  and  to  promote  understanding  of  the  importance  of  consequences,  timing,  risk  and  capabilities 
associated  with  different  events,  behaviours,  and  decision  options.  In  schema  training,  trainees  learn  what 
it  means  to  develop  SA  in  the  environment,  learn  to  build  higher  level  SA  out  of  data,  and  receive  training 
toward  projecting  future  events  in  prototypical  situations. 

SA  Trainer  introduces  SA  using  the  “what”,  “so  what”,  and  “now  what” 
description  model,  which  maps  to  the  three  levels  of  SA  in  a  format  that  is 
easy  to  recall.  In  this  manner,  trainees  learn  to  think  about  their  SA  needs  in 
terms  of  WHAT  information  do  I  have  or  do  I  still  need,  SO  WHAT  does  this 
information  mean  in  terms  of  my  goals,  and  NOW  WHAT  will  happen  next, 
or  what  should  I  be  prepared  to  do  to  deal  with  the  future  situation?  Virtual 
training  instructors  (avatars)  lead  trainees  through  various  situations  to  help 
focus  on  critical  metacognitive  skills  related  to  identifying  critical  cues, 
evaluating  the  relevance  of  information,  both  available  and  missing,  and 
developing  contingency  plans  for  likely  near  future  events.  SA  Trainer  helps 
trainees  learn  how  to  think  about  situations,  rather  than  just  what  to  think,  so 
that  they  quickly  recognize  prototypical  situations,  adapt  to  novel  situations, 
and  make  high  quality  decisions  during  real  operations.  As  trainees  make 
decisions,  the  avatar  may  prompt  trainees  to  consider  what  they  know,  what 
they  need  to  know,  how  they  can  best  gather  missing  information,  and  how  to 
evaluate  conflicting  information. 

The  training  specifically  focuses  on  creating  Levels  2  and  3  SA  from  the  basic  information  available  in  the 
environment.  It  also  focuses  on  training  soldiers  in  the  critical  cues  that  signify  prototypical  classes  of 
situations,  for  instance  leading  indicators  for  IEDs.  This  type  of  training  helps  build  the  mental  models 
and  schema  necessary  to  put  cues  together  to  form  the  needed  comprehension  and  projection.  As  such,  it 
is  experientially  based  (i.e.  involves  mission  scenarios  and  examples)  and  allows  trainees  to  build  an 
understanding  of  cues  and  events  in  a  multi-modal  manner.  A  key  advantage  of  this  type  of  training  is  that 
it  provides  a  concentrated  learning  environment  for  building  up  the  key  foundations  for  SA  that  would 
take  years  to  accumulate  in  the  field  and  would  be  very  hazardous.  In  an  investigation  of  the  efficacy  of 
SA  Trainer,  Strater,  et.  al  [9] found  that  military  cadets  who  had  received  training  with  the  infantry  version 


What?  What  information  do  I 
need?  What  resources  do  I  have 
available?  What  are  the 
environmental  conditions? 

So,  What?  Which  resources 
should  I  task  to  gather  needed 
information?  Which  ones  have  the 
best  capabilities  for  the  task  and 
environment? 

Now  What?  Where  should  I 
position  resources  for  information 
gathering?  Where  are  they  likely  to 
detect  the  desired  target? 


Figure  3.  SA  Trainer  for  Infantry  Operations 
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of  SA  Trainer  performed  better  in  a  field  exercise,  and  were  more  likely  to  have  confidence  in  their  SA. 
In  addition,  we  have  recently  completed  a  version  of  SA  Trainer  specifically  directed  at  training  soldiers  to 
detect  and  defeat  IEDs  for  the  USMC. 

2.2  Virtual  Environment  Situation  Awareness  Review  System  (VESARS) 

Feedback  is  critical  to  the  learning  process.  In  order  to  improve  SA,  soldiers  need  to  receive  feedback  on 
the  quality  of  their  SA.  For  example,  inexperienced  soldiers  may  fail  to  appreciate  the  severity  of 
threatening  conditions  because  they  have  come  through  similar  conditions  in  the  past  just  by  luck. 
Unfortunately,  this  also  reinforces  poor  assessments.  It  is  difficult  for  individuals  to  develop  a  good  gauge 
of  their  own  SA  in  normal  operations.  Training  through  SA  feedback  allows  trainees  to  fine  tune  critical 
behaviours  and  mental  models  based  on  a  review  of  their  SA  performance  [10].  The  VESARS  method 
involves  the  use  of  SA  measures  that  assess  trainee  SA  in  three  areas:  (1)  a  behavioural  rating  tool  that 
assesses  warfighter  and  team  actions,  (2)  a  communications  rating  tool  that  evaluates  team 
communications,  and  (3)  a  SA  query  tool  that  allows  direct  assessment  of  warfighter  and  team  SA  [11, 
12].  VESARS  was  specifically  designed  to  work  well  with  virtual  and  embedded  training  environments, 
or  it  can  also  be  employed  in  field  exercises. 


2.2.1  SA  Behavioural  Rating 

VESARS  presents  aspects  of  SA  behaviour  important  to  acquiring  and  maintaining  SA  for  a  particular  job 
function  or  role.  Instructors  can  rate  individuals  or  teams,  based  on  the  appropriate  behaviours  for  building 
SA  relevant  to  each  job.  This  particularly  supports  the  less  experienced  evaluator,  pointing  out  what  to 
look  for  in  relation  to  physical  actions  for  situation  assessment.  Each  behaviour  can  be  rated  on  a  Likert 
scale  including  ratings  from  1  to  7,  or  “not  observed”.  A  particular  utility  of  the  VESARS  behavioural 
rating  system  lies  in  being  able  to  assess  how  the  quality  of  the  situation  assessment  behaviours  of  the 
trainees  changes  over  time,  as  the  ratings  can  be  made  throughout  the  training  session  and  are  recorded 
along  the  timeline.  This  avoids  the  post-trial  memory  recall  problem  associated  with  making  an  after¬ 
action  assessment  of  all  behaviours  and  the  overgeneralization  that  is  common  to  such  ratings.  The 
VESARS  behavioural  ratings  for  each  team  member  and  each  item  on  the  rating  tools  are  also  averaged  to 
provide  a  behavioural  score  for  the  role  and  for  a  particular  situation  assessment  skill.  The  results  are 
recorded  at  the  end  of  a  training  trial  into  a  data  file  for  later  review. 
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Figure  4.  VESARS  SA  Behavioural  Rating  Form  Example  (USMC  Fire  Support  Team) 


2.2.2  SA  Communications  Rating 

Team  communications  is  critical  to  the  formation  of  SA  in  many  team  environments,  particularly  those  in 
most  military  operations.  Various  team  communications  have  been  categorized  as  relevant  to  SA  on  the 
rating  tool.  Each  item  on  the  rating  scale  has  been  identified  as  an  important  team  communication  for 
acquisition  and  dissemination  of  goal-related  information.  In  using  this  tool,  the  instructor  listens  to  the 
natural  communications  of  the  team  members  and  rates  the  quality  of  their  verbalizations  as  either  good  or 
bad  (poor)  statements.  The  system  tracks  the  counts  of  good  and  poor  statement  over  the  course  of  a 
scenario.  At  the  end  of  a  training  trial,  the  rater  has  the  opportunity  to  review  summary  data  for  SA 
communications  ratings  in  order  to  provide  an  overall  team  communications  rating.  The  rating  scale 
includes  descriptive  phrases  that  serve  as  guidelines  to  raters  when  providing  a  score  from  1  to  5.  The 
descriptions  help  rates  assess  overall  communications  quality  for  consistent  ratings  of  team 
communications . 
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Figure  5.  VESARS  SA  communications  rating  scale  example  (USMC  fire  support  team) 

2.2.3  SA  Probes 

Direct  objective  assessment  of  an  individual's  SA  through  relevant  queries  measures  an  individual’s  SA 
(their  perception  of  what  is  happening  in  a  simulation  or  exercise)  and  provides  feedback  on  how  that 
compares  to  what  was  actually  happening  at  the  time.  This  type  of  information  can  be  highly  useful  in 
teaching  an  individual  the  ways  in  which  their  processes  may  be  deficient  and  how  to  correct  them. 
VESARS  accomplishes  this  goal  through  a  modification  of  the  Situation  Awareness  Global  Assessment 
Technique  (SAG  AT)  [3,  13].  SAG  AT  has  been  adapted  for  SA  training  by  the  addition  of  feedback  to 
individuals  on  the  accuracy  of  their  responses.  For  instance:  “You  thought  you  were  here  when  actually 
you  are  there.  You  had  hostile  individuals  following,  but  were  unaware  of  it.  You  missed  noticing  that 
the  earth  had  been  disturbed  along  that  route,  which  is  a  key  indicator  for  IED  activity”. 

The  VESARS  SA  Probe  tool  supports  the  person  administering  SA  probes  in  identifying  relevant  probes 
over  the  course  of  the  training  scenario,  selecting  and  presenting  probes  in  real  time,  and  recording 
responses  to  probes.  The  comprehensive  list  of  SA  probes  is  developed  for  the  domain  (based  on  a 
detailed  Goal  Directed  Task  Analysis  (GDTA)  conducted  for  each  role)  and  is  housed  in  a  database  that  is 
accessed  by  VESARS.  The  system  scores  each  probe  as  correct  or  incorrect,  retaining  both  the  trainee 
response  and  the  correct,  ground  truth  response  recorded  by  the  administrator.  The  results  are  then  stored 
in  a  data  file  that  can  be  accessed  for  review  using  the  review  tool  to  be  explained  later. 

To  incorporate  shared  SA,  the  GDTA  data  of  all  team  members  are  cross-analyzed  to  highlight  the  shared 
goals  and  shared  SA  needs.  In  many  cases  there  are  similar  informational  elements  that  are  needed  for 
different  purposes  across  team  members.  SA  probes  can  be  developed  to  determine  if  training  team 
members  have  the  same  understanding  of  shared  SA  elements.  The  SA  probe  database  identifies  probes 
that  are  potential  shared  SA  queries,  and  indicates  to  which  team  members  each  SA  probe  is  associated. 
As  some  information  requirements  are  relevant  to  all  members  of  the  team,  SA  probes  designed  to  assess 
knowledge  of  these  information  items  can  be  posed  to  any  member  of  the  team.  Some  information 
requirements  are  only  applicable  to  a  limited  set  of  team  members.  Such  SA  probes  would  be  identified  in 
VESARS  as  only  relevant  to  those  few  positions. 
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Figure  6.  SA  Probe  delivery  tool  example  (USMC  fire  support  team) 


2.2.4  Use  of  VESARs  in  After  Action  Reviews  (AAR) 

SA  training  through  feedback  involves  providing 
knowledge  of  results  immediately  following  training  so 
that  trainees  understand  the  degree  to  which  they  were 
able  to  acquire  SA  and  use  it  to  meet  operational  goals. 
This  kind  of  feedback  enhances  the  utility  of  AAR 
sessions  as  it  provides  quantitative  and  qualitative  results 
that  the  training  coordinator  can  map  directly  back  to 
scenario  events.  This  approach  requires  valid  and  reliable 
measures  of  SA  that  are  based  on  theory  and  empirical 
results.  SA  measures  for  training  and  instructions  should 
meet  specific  criteria  to  ensure  they  provide  “good”  results 
for  “feeding  back”  for  training.  Research-based  criteria  (e.g.,  [14,  15,  16])suggest  that  measurement 
should:  map  to  key  performance  outcomes  and  goals  for  the  domain,  address  both  team  and  individual 
deficiencies,  facilitate  insight  into  team  interactions,  meet  construct  and  content  validity,  be  reliable, 
defensible,  and  relevant,  be  multifaceted,  and  be  practical  for  use  and  implementation.  The  SA  and  human 
performance  measures  developed  for  our  SA-oriented  training  through  feedback  approach  are  designed  to 
meet  all  of  these  criteria  and  to  provide  detailed  and  specific  SA  review.  The  AARs  can  be  conducted 
either  in  a  classroom  setting  following  the  exercise,  or  virtually  through  built-in  AAR  tools  embedded 
within  the  equipment. 

Data  on  the  accuracy  of  each  team  member’s  SA,  their  associated  SA  behaviours  and  SA  communications 
over  the  course  of  the  mission  can  be  presented  as  discussed,  as  well  as  how  these  factors  affected  the 
overall  SA  and  performance  of  the  team.  This  creates  a  detailed  template  for  conducting  the  AAR  and  for 
providing  actionable  feedback  to  the  trainees  to  enhance  the  learning  experience  for  these  critical 
cognitive  skills. 
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Situation  Awareness  (SA)  is  the  perception  of  elements  in  the  environment,  the  comprehension  of  their  meaning  in  terms  of  task 
goals,  and  the  projection  of  their  status  in  the  near  future. 

Percentage  of  Good  SA  Communication  Statements 


HSTlMtfer:  58%  MO  95%  FO  Artlfcrr  55%  FO  M«u«:  47% 


Based  on  the  information  in  the  tables  above,  rate  the  SA  communications  of  the  team  by  selecting  the  option  that  most 
dosely  represents  the  skills  observed  during  the  training  scenario. 

_lJ  _lJ  jlJ 

Haidly  «ny  in  SA  Some  atoll  in  SA  Adequate  skill  *  SA  MghatoRinSA  Complete  otoW  ■>  SA 


Figure  7.  Examples  of  VESARS  Feedback  for  use  in  the  After-action  Review 


VESARS  was  initially  developed  for  the  U.S.  Army  Research  Institute  for  training  small  units  in  urban 
combat  settings,  and  has  been  expanded  to  fire  support  teams  for  the  US  Marine  Corps,  to  homeland 
defence  applications,  and  to  mission  control  training  for  NASA. 

2.3  Situation  Awareness  Virtual  Instructor  (SAVI) 

Training  others  is  a  powerful  opportunity  for  the  trainer  to  learn  as  well  as  for  the  student.  Research  has 
shown  that  acting  in  the  role  of  instructor  also  serves  to  facilitate  learning  [17,  18].  Most  new  trainers 
experience  an  exponential  learning  curve  during  the  initial  transition  from  doing  a  task  to  training  others 
on  a  task.  The  Situation  Awareness  Virtual  Instructor  (SAVI)  trains  warfighters  on  the  behaviours  that  are 
consistent  with  and  important  to  good  SA  by  allowing  trainees  to  play  the  role  of  the  trainer  to  rate  the 
actions  (behaviours  and  communications)  of  an  individual  or  team  and  provide  a  rationale  for  their  rating. 
SAVI  integrates  behavioural  and  communications  training  through  embedded  training  technologies  so  that 
warfighters  can  observe  performance  and  understand  how  information  gathering  and  dissemination 
components  (e.g.,  attention  and  perception,  information  discovery  and  filtering,  information  prioritization 
and  integration,  use  of  operational  tools,  information  exchange  and  provision  contextual  content,  team 
coordination,  etc.)  interact  to  impact  global  SA.  The  objective  behind  this  training  method  is  that  the 
process  of  critically  evaluating  the  behaviours  and  communications  portrayed  in  the  videos  allows  trainees 
to  gain  a  deeper  understanding  of  how  these  behaviours  and  communications  support  the  acquisition  of 
superior  SA. 
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The  SAVI  approach  leverages  the  exponential  learning  that  occurs  during  either  peer  instruction  or  the 
transition  from  trainee  to  trainer.  Trainees  quickly  learn  what  behaviours  are  appropriate  for  various 
operational  situations,  because  they  observe  these  aspects  of  performance  and  provide  their  assessments 
on  the  quality  of  the  performance.  Trainees  are  able  to  refine  their  mental  models  of  good  SA  behaviours 
and  communications  as  they  also  compare  their  assessments  to  those  provided  by  domain  experts. 
Behavioural  training  facilitates  understanding  of  how  active  situation  assessment  promotes  SA  for 
maintaining  the  tactical  advantage. 


Figure  8.  Examples  of  SAVI  for  Mission  Control 


3.0  CONCLUSION 

Taking  advantage  of  the  capabilities  of  embedded  training  tools  involves  far  more  than  translating  the 
same  old  training  techniques  to  a  new  platform.  The  real  advantage  will  be  seen  when  we  move  from 
simply  training  the  rote  skills,  towards  training  the  cognitive  skills  that  allow  warfighters  to  utilize  their 
equipment  and  techniques  in  situ  to  achieve  the  best  performance  for  the  situation  at  hand.  As  much  more 
sophisticated  command  and  control  tools  are  developed,  the  cognitive  aspects  of  warfighting  will  become 
even  more  critical  than  they  have  been  in  the  past.  In  this  paper  we  advance  several  techniques  and  tools 
that  have  been  developed  for  building  the  cognitive  skills  that  will  be  needed  on  tomorrow’s  battlefields. 
These  tools  can  be  incorporated  into  embedded  training  programs  to  allow  warfighters  to  develop  and 
hone  the  needed  SA  skills  for  achieving  high  levels  of  performance. 
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